Skip to content

Add native celld KV through an EdgeZero store service on stock celld - #422

Open
ChristianPavilonis wants to merge 22 commits into
issue-410-native-store-providersfrom
feat/native-celld-stores
Open

ChristianPavilonis wants to merge 22 commits into
issue-410-native-store-providersfrom
feat/native-celld-stores

Conversation

@ChristianPavilonis

@ChristianPavilonis ChristianPavilonis commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Current state: KV on official celld, no modified build

Head 15bdeef6. This PR no longer depends on a modified celld. The scoped listener compiled into celld is gone; live KV now goes through an EdgeZero-owned store service that the official celld release runs as an ordinary application.

Why. Stock celld only exposes storage to Worker code deployed into it, so EdgeZero ships that code: services/celld-store, a Worker plus one SQLite Durable Object per namespace, deployed with celld deploy. Spec: docs/superpowers/specs/2026-11-16-stock-celld-store-service-design.md.

What changed in this update

  • New services/celld-store/ (single module, no_bundle, no npm dependencies). Every mutation applies in one SQLite transaction and records the client's mutation ID. Stock celld resends a buffered Durable Object call after a pooled peer tunnel fails, even if the owner already ran it (send_inner in crates/celld/main/peer_tunnel.rs, v0.6.3); the ledger answers that resend with the first outcome. An apply-before window stops a call held through cell recovery from applying long after the client gave up.
  • New wire protocol: x-edgezero-store-version: 1, a store instance header on every object response, mutations as POST /put and POST /delete with x-edgezero-mutation-id. Forked servers and reset stores fail instead of answering. The client still never resends.
  • Removed the celld-config provider and the fork-only v1 record API (records feature). Records will come back on this service through the record-contract work.
  • Replaced the modified-celld runner and namespace-isolation script with proofs against the official release.
  • Docs: guide rewritten, including an automated Kubernetes deploy Job so operators have no manual step.

For review: each namespace is one Durable Object. Every write for a namespace is serialized on one node and waits for durability, that node carries all of that namespace's traffic, a node failure pauses the namespace until takeover (10 s lease by default), and sharding later would be a storage format change.

Evidence (official celld v0.6.3, binary SHA-256 eb0a8158421b69d68dd5d28f2298a7d411dde498295833deeb1cc5ce53b7eb75, release attestation verified):

  • node --test service, runner and fleet-script suites: 17, 9 and 4 passed.
  • edgezero-store-celld settings / kv / all-features: 4 / 32 / 32 passed. Axum native-bindings lib 25, celld-kv startup 4, mixed AWS+celld lib/startup 442 + 5 + 4 passed.
  • cargo clippy --workspace --all-targets --all-features -D warnings, focused feature-combo Clippy, cargo fmt --check, strict rustdoc for the changed crates, dependency isolation script.
  • Generated consumer, deterministic fixture: default-off rejection, live KV, one unique mutation ID per write, restart, AWS coexistence, delete.
  • node scripts/run_native_celld_integration.mjs: generated consumer against celld dev running the service behind a TLS proxy, authenticated. Binary and empty values, listing, 61-second TTL expiry, restart, rejected token.
  • node scripts/run_celld_store_fleet_proof.mjs: RustFS 1.0.1 container, two nodes. A reply dropped after the owner ran a put made celld resend it; the client got 204 and the value applied once. kill -9 of the owner, then takeover: all acknowledged writes present.
  • Docs format, lint and build.
  • Hosted CI on this head.
  • Production bucket, network isolation, adjacent-version behavior during in-place deploys, sustained throughput. Local loopback numbers only: about 63 sequential writes/s, about 240 writes/s at concurrency 32 on one namespace.

Not done here: forward merges into #439 and the rest of the stack, record endpoints on the service, Trusted Server #1537 fixture changes, new RC tags, and the #437 issue text (it still lists config).

Previous description (modified-celld design, superseded)

Current review readiness

Combined head: 1c90b246af22dad4e38899c8a12a0f81cb2c6fed. This PR includes native config snapshots, ordinary KV and optional conditional/physical record operations. The records feature remains default-off and requires the additive celld record-server extension. Ordinary KV does not acquire CAS semantics.

Local checks pass on the combined tree:

  • Workspace tests: 2,885 passed, 0 failed, 6 ignored; workspace strict all-feature Clippy, formatting, feature compilation and Spin wasm32-wasip2 compilation pass.
  • Provider default/config/KV/records/all-feature suites: 5 / 28 / 31 / 48 / 56 passed; scoped strict Clippy and dependency isolation pass.
  • Four celld startup tests, the explicitly selected generated-consumer fixture and the explicitly selected real conditional-record interop test pass. The real-service test used the previously fingerprinted modified celld binary with disposable loopback data, not an independent service rebuild or fleet qualification.
  • The changed guide passes Prettier 3.9.10.

Fresh hosted CI and review of the enlarged scope are pending. Existing review feedback remains open; earlier approvals and receipts do not cover the added record feature. Historical checkpoints below describe only their recorded inputs. No deployment, release-tag update or main merge is included.

Current review-stack checkpoint

Publication head: f9b0f20c6857d2e638bcfe7bf1edaa5af5425bfb. This cleanup preserves the existing feature unit and PR identity. The additional change is the completed generated-native build correction: image-only code now follows the scaffold's existing Fastly-example selection, with no request-policy change. Its focused generator/native proof is retained; fresh per-branch hosted checks are not assumed green.

Review order: #275 → #406 → #407 → #409 → #414 → #422 → #439 → #454. Draft integration checkpoint: #462. New SDK/group/codec/phase development and unrelated TS work are not prerequisites or additions to this PR. Earlier notes below retain their original tested revisions and limits; they are not a whole-tree pass for this new head.

Earlier implementation and proof notes

RC refresh checkpoint, 2026-11-10

Current head: d475727691e2a80ca2dbb324568283f63e34a870.

The five celld-only donor commits are replayed onto updated #414, without obsolete hosting or diagnostic ancestry. Config/KV source selection stays typed and feature-isolated; startup snapshots are bounded and KV preserves exact keys, prefixes, continuation and expiry rules.

Feature proof passes 38 celld cases, independent settings/config/KV suites, 478 Axum unit, 6 mixed AWS startup and 4 celld startup cases, CLI/isolation/runner checks and scoped strict Clippy. Independent provider source review found no confirmed defect. Lower namespace/CI fixes are merged and all five locked fetches pass.

Ordinary KV is not conditional create/CAS/atomic RMW or identity-allocation qualification. Optional conditional/physical records provide separate Add, Overwrite and generation-match operations without changing ordinary KV. Prepared celld KV does not prove runtime connectivity/grants, and snapshots are not atomic across documents. Production identity-allocation and fleet acceptance remain separate gates.

Draft pending fresh hosted checks and the combined RC audit. Two unchanged native default-parallel deadline misses remain recorded and unclassified; serial passes and untouched-foundation parallel passes do not clear them. No timing bound was relaxed. No deployment, main merge, security-alert dismissal or review-thread resolution occurred. Historical design/review context follows.


Summary

  • Add native celld configuration snapshots, live KV and conditional/physical records through a scoped API embedded inside celld, with default-off config/KV/records features.
  • Preserve portable store traits and native startup ordering, with separate AWS/celld policies, credentials, accounting and actual-variant preparation sessions.
  • Add bounded transport/fault tests, public-runner and generated-consumer proofs, structural-only CLI support, documentation and CI checks.

Changes

Crate / File Change
edgezero-store-celld Strict settings, redacted explicit tokens, bounded HTTP/TLS client, immutable config snapshots, live KV and optional conditional/physical records. No celld runtime or V8 dependency.
edgezero-adapter-axum Per-ID typed bindings, once-only selected token capture, independent provider policies/capabilities and existing readiness/staging integration.
edgezero-cli and generated Axum manifest Independent feature forwarding, structural-only validation with availability unknown, and selected remote config-push refusal.
Tests, scripts and CI Feature-isolation checks, TLS/proxy/no-replay fault tests, mixed AWS/celld/local startup, generated consumers and explicit local-service acceptance.
Consumer lockfiles Refresh settings-only provider edges for the excluded demo and reusable-app workspaces without registry version changes.
Documentation Native celld guide, reviewed spec/plan and exact local verification evidence with remaining gaps.

Closes

Closes #437.

Related to #410 and stacked on #414. This is store-provider work, not the celld HTTP runtime adapter in #421.

Dependencies and draft status

  • Base branch: issue-410-native-store-providers, currently draft Add native AWS store providers #414.
  • Config and ordinary KV require the unreleased embedded celld scoped API at f72ae21a515859570622ac9d6ca9c19cf6870b81. Records additionally require the conditional-record server extension with x-celld-record-version: 1; the original listener does not support records. The companion celld changes are not published by this PR.
  • No gateway, sidecar, Worker bridge, celld secret provider or portable core trait changes.
  • Fleet forwarding/reload/failover, production deployment isolation, replication/storage acceptance, upgrades and sustained fault/shutdown stress remain open. The generated TLS and disposable namespace-isolation proofs are separate arrangements.
  • A real-service CI job remains deferred until a reviewed service artifact is available. Missing prerequisites fail the explicit local harness instead of passing by skipping.

Test plan

  • cargo test --workspace --all-targets: 2,028 passed, 6 ignored. Ignored acceptance tests listed below were explicitly selected separately.
  • cargo clippy --workspace --all-targets --all-features -- -D warnings, with focused Clippy follow-ups for the final test-only additions.
  • cargo fmt --all -- --check and git diff --check.
  • cargo check --workspace --all-targets --features "fastly cloudflare spin".
  • All provider WASM target builds. Spin wasm32-wasip2 check passed; full Fastly and Cloudflare target builds were not executed.
  • examples/app-demo full workspace tests. Both excluded consumer lockfiles passed offline locked metadata validation; reusable fixture-axum check passed.
  • Docs lint, format and site build.
  • Manual edgezero serve --adapter axum. Production-runner and generated native binaries were exercised instead.
  • Provider settings/config/KV/combined suites: 4 / 27 / 30 / 38 passed, with independent feature Clippy checks.
  • Native feature matrices, 253 combined native units, 4 celld startup subprocess tests, and 6 mixed/AWS startup tests. AWS-only startup also passed all 5 existing cases.
  • Generated celld target with neither feature, each feature alone, both, and both alongside all AWS provider features.
  • Real generated app against modified celld over verified TLS: binary/empty KV, pagination, 61-second expiry, immutable config adoption and persistence across app/celld restarts.
  • Disposable user/network/mount/PID namespace proof for scoped access, internal-listener denial, hidden service files and server procfs isolation.
  • Existing generated AWS and DynamoDB proofs, dependency feature-isolation checks, and 24 integration-runner Node tests.

All Cargo verification used offline locked dependencies after reviewed lockfile updates. The workspace test runner needed stdin closed because existing consent tests assume noninteractive CI. No test was excluded to resolve that run.

The real-service command requires an explicit modified binary and uses only disposable state and dummy credentials:

CELLD_BIN=/absolute/path/to/modified/celld \
EDGEZERO_GENERATED_TEST_TARGET_DIR=/absolute/path/to/disposable-build-cache \
node scripts/run_native_celld_integration.mjs

The implementation plan records the tested source/binary fingerprints and proof limits. No real AWS/fleet credentials or production deployment were used.

Checklist

  • Changes follow CLAUDE.md conventions.
  • No new direct Tokio dependencies in core or adapter crates; native transport lives in the provider crate.
  • No legacy route-param syntax introduced.
  • Application handlers use edgezero_core types.
  • Store wiring uses the existing typed registries and staging path.
  • New behavior has tests.
  • No production secrets or credentials committed. The checked-in DER key/certificates and token strings are public, disposable fixture material only.

Owned follow-up review checkpoint

  • Branch advanced through an ordinary merge; every previous public head remains an ancestor.
  • Current head: ef1d4ab59427642ef26b0a30d679a068f040157e. Source tree 43174231bb50d19072f33ed7a19734b5bae8c966 is byte-identical to previous head bb56cfe142965bea6c997d0d2e288e2698f8ee64.
  • Related owned draft: #499. It is intentionally a placeholder until the author cleans feat: implement portable outbound HTTP and request lifecycle contracts #275 and the owned baseline/follow-up are reconciled.
  • Review base and ready/draft state are unchanged. The feature-only review diff is byte-identical to the pre-update diff.
  • Existing qualification remains tied to its recorded inputs. New hosted outcomes are reported separately; this history update does not qualify new source-version-bound binaries or move a release tag.

@prk-Jr prk-Jr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

Summary

Reviewed head bb56cfe1 against the PR base (issue-410-native-store-providers @ 3733482f). The commits since the previous reviews (on f9b0f20c) only merge base-stack changes and touch none of this PR's 43 files, so every earlier finding still applies unchanged. The blocking items from @aram356's review are still open and have no author replies. That is the reason for requesting changes here; this review adds no new blocking defect of its own.

😃 Praise

  • Endpoint parsing is strict about origins: it rejects 127.1, decimal-integer IPs, localhost, trailing dots, credentials in the URL and path segments, and it requires HTTPS for authenticated access (settings.rs:231).

Findings

Blocking

  • 🔧 More unguarded celld test steps (test.yml:201-205): the same run_test_nonzero.sh gap as the existing startup-suite comment, on the five --lib steps. Posted as a reply on that thread.

Non-blocking

  • ⛏ An empty [limits] table is now silently accepted (native_bindings.rs:147)
  • 🤔 Two isolation probes can never fail (check_celld_scoped_isolation.sh:95, :103)
  • ⛏ The timeout test doesn't prove the process group is killed (run_native_celld_integration.test.mjs:101)
  • ⛏ Guide: setting limits in both places is a startup error (native-celld-stores.md:160)
  • ⛏ Plan doc: wrong lockfile path and personal home-directory paths (plan :493, :34)

CI Status (head bb56cfe1)

  • Hosted: everything passes except cache-new-revision-smoke and cache-restore-smoke (Deploy actions). Both fail on a cache miss (TARGET_CACHE_HIT: false). This PR doesn't touch that workflow, and the same jobs passed on #414 in the same batch, so this looks like a cache/infrastructure problem rather than this PR.
  • Local:
    • axum lib tests across two feature sets pass.
    • Node runner tests pass (9/9).
    • Single-feature celld clippy with -D warnings on 1.98.1 is clean.
    • --locked metadata passes for all three lockfiles.
    • The guide's TOML examples pass config validate as documented.
  • Not run locally: the real-celld TLS proof and the namespace isolation script (they need the modified celld binary and Linux).

#[serde(deny_unknown_fields)]
struct LimitsTable {
aws: AwsPreparationLimits,
aws: Option<AwsPreparationLimits>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛏ An empty [limits] table is now silently accepted

On the base branch LimitsTable.aws was required. A bare [limits] header with no [limits.aws] was rejected as malformed, and a [limits] table combined with an explicit policy was rejected as Duplicate (self.limits.is_some()).

With both fields now Option, an empty [limits] (for example a typo, or keys placed under the wrong section) parses as LimitsTable { aws: None, celld: None }. Both providers then quietly use their default policies. The duplicate check (:351-357) now looks at each provider separately, so an empty table plus an explicit policy is also accepted.

Nothing becomes unsafe, but an operator mistake is ignored instead of reported.

Fix: in validate_with_policies, reject Some(LimitsTable { aws: None, celld: None }) as malformed.

Comment thread scripts/check_celld_scoped_isolation.sh Outdated
[[ $(request -X PUT --data-binary forbidden "$base/v1/stores/config/value?key=probe") == 403 ]]
[[ $(request "$base/v1/stores/undeclared/value?key=probe") == 403 ]]
[[ $(request "$base/shutdown") == 404 ]]
if curl --noproxy '*' --max-time 1 -sf "http://10.203.0.1:$internal/state" >/dev/null; then

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 This probe can't fail, whatever the isolation

celld starts its internal listener on --internal-listen 127.0.0.1:0 (line 38), and the port is parsed from celld internal listening on 127.0.0.1:... (line 53). Nothing ever listens on 10.203.0.1:$internal, so this curl fails whether or not the namespaces isolate anything. Only the loopback probe at line 98 and the private PID namespace actually test isolation.

The procfs loop (lines 103-107) has a similar problem: it matches only comm == celld, so a CELLD_BIN with any other basename (for example celld-scoped) passes automatically.

CI doesn't run this script, so this is non-blocking. The plan and guide do cite its result as isolation evidence, though.

Suggestion: add a control step that shows the probed address is reachable from a non-isolated namespace (or bind the internal listener to the veth address for that check), and match the server by PID or executable path rather than by name.

"const fs=require('node:fs');",
"const child=spawn(process.execPath,['-e','setInterval(()=>{},1000)'],{stdio:'ignore'});",
"fs.writeFileSync(process.argv[1],String(child.pid));",
"process.on('SIGTERM',()=>{child.once('exit',()=>process.exit(0));child.kill('SIGTERM');});",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛏ This test passes even if the runner signals only the direct child

The runner's guarantee is that it kills the whole process group (process.kill(-child.pid, signal)), so no descendant survives. Here, though, the fixture parent handles SIGTERM by forwarding it to its own child and waiting for it to exit. If kill regressed to child.kill(signal), the grandchild would still be reaped and the test would still pass.

Fix: have the fixture parent exit on SIGTERM without forwarding the signal (process.on('SIGTERM',()=>process.exit(0))). Then only the group kill can reap the grandchild.

Comment thread docs/guide/native-celld-stores.md Outdated
| Store alias | At most 64 ASCII letters, digits, underscores or hyphens |
| Preparation / operation / connect timeout | 30 / 5 / 1 seconds |

`[limits.celld]` or `with_celld_preparation_limits` selects one celld policy. File and explicit policies do not merge. AWS has an independent policy and accounting. Client caches hold at most 128 authorization-scoped entries. Accounting describes logical payload and metadata quotas, not process RSS.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛏ "File and explicit policies do not merge" reads as though one of them simply wins. In fact, supplying both [limits.celld] and with_celld_preparation_limits fails startup with Duplicate (native_bindings.rs:351-357), and so does a second explicit call (run_options.rs:568-570). Please say that setting both is a startup error.

The guide also never names the [limits.celld] keys (max_value_bytes, operation_timeout_ms, ...). Listing them next to the limits table would help.


The user subsequently authorized committing the EdgeZero changes and opening a draft PR. The PR is stacked on #414, targeting `issue-410-native-store-providers`, not main. The companion celld branch remains local and is a separate publication dependency.

Pre-publication locked metadata checks found stale `examples/app-demo/Cargo.lock` and `tests/fixtures/reusable-app/Cargo.lock`. Both were refreshed offline with only the new settings-only provider package and adapter edge added. No existing registry versions changed. Both locked graphs then passed, and `cargo check --offline --locked --manifest-path tests/fixtures/reusable-app/Cargo.toml -p fixture-axum` passed in 2.78 seconds using the shared native target directory. Full demo workspace tests and all three provider WASM target builds remain unexecuted for this publication step. These consumer lockfiles are outside the source fingerprint above.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛏ This cites tests/fixtures/reusable-app/Cargo.lock and --manifest-path tests/fixtures/reusable-app/Cargo.toml. The real location is crates/edgezero-test-harness/fixtures/reusable-app/, so the recorded command can't be rerun as written.

Separately, lines 34, 105, 391 and 395 embed personal /home/pav/worktrees/... paths, which are worth dropping from a shared doc.

}
}

pub(crate) fn endpoint(value: &str, access: &CelldAccess) -> Result<Url, PreparationError> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😃 Strict origin parsing. Comparing the raw host with the parsed IP rejects 127.1 and decimal-integer forms. It also rejects localhost, trailing dots, userinfo and path segments, and it refuses plaintext HTTP for authenticated access. That closes off a whole class of SSRF and confusion tricks before any network I/O.

@ChristianPavilonis ChristianPavilonis changed the title Add native celld config and KV store providers Add native celld config, KV and conditional record providers Nov 13, 2026
aram356 added a commit that referenced this pull request Nov 14, 2026
Brings #275 at e9ffaa8 and the refreshed heads of #499, #460, #406,
#407, #409, #414, #422 and #439. Each PR merged the one below it, so
this merge has no conflicts.

Two edits fit #469 to #275, as #469's open review asks:

- #469's ingress JA4 test calls into_core_request with one argument.
  #275 now passes the Axum outbound executor through that function, so
  the test gets an executor handle the way the other ingress tests do.
- The Axum guide names EdgeZeroAxumService, which #275 removes and
  check_outbound_legacy_api.sh rejects. The guide now names only
  AxumDevServer::with_trusted_ja4_header.
aram356 added a commit that referenced this pull request Nov 16, 2026
Brings #275 at b65e714 and the refreshed heads of #499, #460, #406,
#407, #409, #414, #422 and #439. Each PR merged the one below it, so
this merge has no conflicts. #409 resolved the one conflict on the way
up, in edgezero-core's app.rs imports, by keeping both.
…ck celld

The celld providers depended on a scoped listener compiled into a modified
celld. Stock celld only exposes storage to Worker code deployed into it, so
EdgeZero now ships that code: services/celld-store, a Worker plus one SQLite
Durable Object per namespace, deployed with `celld deploy` onto the official
release.

- Each mutation applies in one SQLite transaction and records its client
  mutation ID. Stock celld resends a buffered Durable Object call after a
  pooled peer tunnel fails even when the owner already ran it; the ledger
  answers that resend with the first outcome. An apply-before window keeps a
  call held through cell recovery from applying long after the client gave up.
- Responses carry a new protocol header and a store instance ID, so forked
  servers and reset stores fail instead of answering.
- Mutations are POST /put and /delete with a fresh 128-bit ID; the client
  still never resends.
- Remove the celld config provider and the fork-only v1 record API.
- Replace the modified-celld runner and namespace isolation script with a
  generated-consumer proof against official celld behind a TLS proxy, and a
  two-node fleet proof that reproduces the resend and an owner takeover.
… tokens

Review follow-ups for the store service:

- A 404 counts as an absent key only when the store instance header is
  present. A routing 404, such as from a proxy that rewrote the path, is now
  malformed instead of silently reporting existing keys missing.
- The Worker accepts any ASCII graphic token, including commas, and still
  rejects repeated Authorization headers by grammar.
- The Kubernetes deploy recipe maps index.js back under src/, because
  ConfigMap keys cannot contain "/".
- The spec no longer claims to detect a rollback to an older snapshot of the
  same store.
@ChristianPavilonis ChristianPavilonis changed the title Add native celld config, KV and conditional record providers Add native celld KV through an EdgeZero store service on stock celld Nov 16, 2026
aram356 added a commit that referenced this pull request Nov 17, 2026
Brings #275 at ea2549f and the refreshed heads of #499, #460, #406,
#407, #409, #414, #422 and #439, including #422's stock-celld store
service and its merge into #439. Each PR merged the one below it, with
no conflicts on the way up, so this merge has no conflicts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add native celld configuration and KV store providers

4 participants